Fast, Accurate, and Practical Identity Inference Using TV Remote Controls

Non-invasive identity inference in the home environment is a very challenging problem. A practical solution to the problem could have far reaching implications in many industries, such as home entertainment. In this work, we consider the problem of identity inference using a TV remote control. In particular, we address two challenges that have so far prevented the work of Chang et al. (2009) from being applied in a home entertainment system. First, we show how to learn the patterns of TV remote controls incrementally and online. Second, we generalize our results to partially labeled data. To achieve our goal, we use state-of-the-art methods for max-margin learning and online convex programming. Our solution is efficient, runs in real time, and comes with theoretical guarantees. It performs well in practice and we demonstrate this on 4 datasets of 2 to 4 people. Introduction Providing multimedia content in a personalized TV environment that aligns the most with the interests of its consumers is a challenging problem for both service providers and content developers. This problem becomes even more challenging in families, where the recognition of individual members is highly desirable. The goal is to provide the best personalized experience for various multimedia contents, such as TV, on-demand programming, interactive media, targeted advertising, online gaming, and many others. In our paper, we discuss practical challenges in building a non-invasive system for identifying TV viewers. The system minimizes the need of the TV viewers to log into their entertainment profiles, or being identified by an invasive method, Copyright c © 2010, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. such as a camera combined with face recognition. Our solution is designed as follows. Whenever the TV viewer reveals identity, we use the corresponding remote control data, such as a sequence of button presses and accelerometer readings, to train the predictor of the person. When the identity of the viewer is unknown, we infer the identity based on the remote control data. Although our solution is specific to the remote control domain, note that many ideas in the paper generalize beyond it. In particular, our work is a prime example of how to turn an offline and fully-supervised solution into an online solution on partially labeled data. The basis of our engineering efforts relies on the paper of Chang et al. (2009), which employs support vector machines (SVMs) and max-margin Markov networks (MNs) (Taskar, Guestrin, and Koller 2004) to infer the identify of TV remote control users. We address two main problems that had so far prevented this work from being implemented in a home entertainment system. First, the original methodology assumes completely labeled data. Unfortunately, TV viewers usually provide very little feedback about their identity. Second, the approach of Chang et al. (2009) does not allow for an incremental improvement of learned predictors. This is necessary since remote control data are usually unavailable in advance, and only become sporadically available as time progresses. Considering our minimal invasive system setup, in which we rarely observe labeled data, we show that there is simply not enough data to build a reasonably good manifold, which can be used by semi-supervised learning algorithms. In turn, we focus on supervised learning only and try to evaluate how many labeled examples are needed to learn good predictors. First, we answer this question in the offline setting. Second, we show how to learn online from completely labeled data. Household Participants Sessions 1 4 458 2 2 124 3 3 28 4 2 90 5 4 340 Table 1: Households statistics. Finally, we relax the assumption on completely labeled data. One of our results is that the identity of remote control users can be inferred with an acceptably high accuracy even when only 20 percent of data are labeled. These results are obtained using state-of-the-art methods for online convex programming (Zinkevich 2003). Although our solution is simple and learned online, the accuracy of the solution is often comparable to Chang et al. (2009). To further improve the solution, we propose a new way of training max-margin Markov networks online (Ratliff, Bagnell, and Zinkevich 2007). The algorithm runs in real time and can be implemented on a commercial platform. Finally, both of our solutions are comprehensively evaluated on 4 remote control datasets of 2 to 4 people, and compared to online and offline majority class baselines. The following notation is used in the paper. The symbols xt and yt ∈ {−1, 1} denote the t-th data point and its label, respectively. The data points xt are divided into labeled and unlabeled sets, l and u, and the labels yt are observed for the labeled set only. The cardinality of the labeled and unlabeled sets is nl = |l| and nu = |u|, respectively, and T = nl + nu denotes the total number of training examples. Remote control dataset Our data set consists of data collected on five households for a period of one to three weeks in which the number of users for the households varied between two and four. This is the same dataset featured in the Chang et al. (2009) paper. Each household had a data collection system that consisted of a tri-axis accelerometer attached to a TV remote control, the corresponding accelerometer receiver, a universal infrared receiver to capture button presses from the remote control, and a laptop to which the receivers were connected. The laptop logged and time stamped the data from the sensors using 100 nanoseconds resolution. For our analysis, all data of interest revolved around the remote control activity in the form of button press selections. Since we seek to associate combined accelerometer and button readings to individual users, we concern ourselves with the behaviors just before and just after each of the button presses. To study just how much before and how much after, we implement four different capture windows at 0.5, 1, 2, 4 seconds with respect to the button press. The windows were designed to help capture the hand motions preceding, centered, and succeeding each button press. We believe the variety in window sizes is sufficient to capture the uniqueness in hand motions for each of the users. Since data was collected using 100 nanoseconds resolution, the data contained within each of those windows were used to generate the features that defined each button press instance. Some of the features associated with the accelerometer data include: energy, fundamental frequency, range, mean, and variance for each of the axes, as well as correlation among each pair of axes. Some of the features associated with the infrared button signal from the remote control include the button code, press duration and number of times the button code was sequentially transmitted. In all, a total of 372 combined features defined each button press instance. The time stamped instances were then group into sessions. A session represents the periods of time in which there is continuous remote control activity. A session ends when it is determined that the remote control still idle. Table 1 illustrates the total number of sessions for each household. Notice that because of the scarcity of sessions, Household 3 was not considered in this study. For our experiments, all the instances corresponding to the same session are aggregated into a session-level instance representation as the mean for the session. Button that were rarely use were discarded before aggregation. We then normalize with respect to all sessions.